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1 his thesis examines the initial decision process of organizing to convert a course 
module from the Navy Medical Department Executive Management Education. The 
objective is to track and model the process used to integrate Network-Based Instruction 
technology into an existing traditional classroom course of instruction. This research 
include survey of Network-Based Instruction literature, and an assessment of the decision 
process involved. The goal is to capture the context of the key decisions made during the 
preliminary stage of an actual conversion project, analyze the effectiveness of the approach 
and if possible generate a model for future efforts. 
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I. INTRODUCTION 



I his research will examine the decision process behind the conversion of a course 
module from the Navy Medical Department Executive Management Education (EME) 
curriculum to Network Base Instruction (NBI). The objective is to track and model the 
process used to integrate NBI technology into an existing traditional classroom course of 
instruction. Research will include a survey of literature on NBI technology, and an 
assessment of the decision process involved. The goal is to capture the context of the key 
decisions made during the preliminary stage of an actual conversion project, analyze the 
effectiveness of the approach, and, if possible, generate a model for future conversion 
efforts. 

A. RESEARCH QUESTIONS 

Primary Research Question: 

1 . What process was used to transform the current EME module to a NBI version? 
Subsidiary Research Questions: 

1 . Who were the stakeholders? 

2. What was the perceived value of the conversion to NBI? 

3. What were the significant resources and constraints on this conversion project? 

B. SCOPE 

The scope will include: (1) tracking the decision process in the preliminary stage of 
converting an existing EME course module to NBI, (2) evaluating the process of converting 
current EME course module to NBI, and (3) reviewing costs associated with changing the 
process. The thesis will conclude with a proposed template for the process of converting a 
course module to NBI. 
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c. 



METHODOLOGY 



The methodology used in this thesis research consists of the following steps: (1) 
Conduct a literature search of applicable Department of Defense (DoD) instructions, 
notices, studies, command policies, books, magazine articles, and other information relevant 
to EME and NBI; (2) Conduct interviews with key players at the Bureau of Medicine and 
Surgery (BUMED), Naval Postgraduate School (NPS), and vendors; (3) Track and model 
the process used to integrate NBI technology into an existing EME course module; (4) 
Prepare an assessment of the decision process used to integrate NBI technology into EME 
course module, (5) Identify potential cost savings from changes to the existing process. 

D. BENEFITS OF THE STUDY 

This study will provide the information required to better manage the conversion 
process of an existing EME curriculum to NBI. 

E. CHAPTER OUTLINE 

Chapter I discusses the purpose, scope, methodology, and benefits of the study. 
Chapter II provides the background about the Congressional mandate, initial DoD and Navy 
administrative health care assessment, the NPS Executive Management Education Program, 
and EME Stakeholders. In addition, the chapter provides a literature review of Network 
Based Instruction. Chapter III describes the decision processes used in the conversion of an 
EME course module to NBI. Chapter IV presents an assessment of the decision process 
utilized. Chapter V provides conclusions and recommendations based on the analysis 
presented in Chapter IV. 
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II. BACKGROUND 



A. CONGRESSIONAL MANDATE 

The U.S. Congress, in 1992 and 1996, issued specific directions to the Department 
of Defense (DoD) concerning the preparation of officers to command military medical 
treatment facilities (MTF) and to serve as TRICARE lead agents or key members of the 
lead agent staff. This initiative continues today to meet its obligations in preparing those 
officers for their duties. 

To direct an MTF in the fulfillment of its mission, the effective commander must 
possess a mixture of specific skills, knowledge, and experience. This mix produces 
competencies that vary with the education and life experiences of the individual. The need 
for specific competencies is particularly evident for MTF commanders, lead agents, and 
their key staff members, who must have both general management skills and a thorough 
understanding of the health care delivery process. Commanders must successfully manage 
the various types of complex organizations that exist in the DoD health care environment. 

The commander of an MTF and the lead agent of a TRIC ARE region are in unique 
positions when compared to their civilian peers. In addition to having health care 
administration skills similar to those in the civilian sector, they must also fulfill military 
requirements. Military requirements include readiness preparation, deployment skills, 
support for war fighting, direction of joint military and civilian health care networks, and a 
number of other requirements. Therefore, the necessary skills and knowledge of an MTF 
commander or lead agent not only must match, but also must exceed, those of their civilian 
counterparts. 

The specific needs of MTF commanders were emphasized by the DoD 
Appropriations Act of 1992, Section 8096, which stated: “None of the funds appropriated in 
this Act may be used to fill the commander’s position at any military treatment facility with 
a health care professional unless the prospective candidate can demonstrate professional 
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administrative skills." Similarly, the 1996 Defense Authorization Act (Public Law 104- 
106. Title VII. Subtitle B. Section 715), which focuses on training in health care 
management and administration for TRICARE lead agents states: 

(a) PROVISIONS OF TRAINING. Not later than six months after the date 
of the enactment of this Act, the Secretary of Defense shall implement a 
professional educational program to provide appropriate training in health 
care management and administration. 

(1) to each commander of a military medical treatment facility of the DoD 
who is selected to serve as a lead agent to coordinate the delivery of health 
care by military and civilian providers under the TRICARE program; and 

(2) to appropriate members of the support staff of the treatment facility who 
will be responsible for daily operations of the TRICARE program. 

(b) REPORT ON IMPLEMENTATION. Not later than six months after the 
date of the enactment of this Act, the Secretary of Defense shall submit to 
Congress a report describing the professional educational program 
implemented pursuant to this section. [Ref. 1 ] 



B. INITIAL DOD ASSESSMENTS 

In December 1991, the Assistant Secretary of Defense, Health Affairs, in response 
to the 1992 Appropriations Act, convened a working group representing the Army, Air 
Force, Navy, and Coast Guard to review and identify professional administrative skills 
necessary for command of an MTF. The group reviewed several areas, including the 
operation of civilian and military hospitals, the responsibilities of senior leaders in both 
settings, service programs in place to prepare individuals for command, and civilian 
standards for successful leadership in a health care organization. They identified the skills 
and experience necessary for successful job performance, discussed alternatives, and 
determined documentation requirements. Additionally, they identified and described 34 
competencies needed to command an MTF, along with suggested procedures for validating 
candidates’ possession of these competencies during the command selection process. [Ref. 
1 ] 
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c. 



THE NAVY INITIATIVE 



The Navy's Bureau of Medicine and Surgery (BUMED), in partnership with the 
Naval Postgraduate School, has designed a program based on the competencies identified in 
the studies cited. The program. Executive Management Education (EME), is designed to 
meet the requirements of the Appropriation Act. The EME Program is being offered at 
different MTF sites and executive sessions at NPS, Monterey, California. 

During extensive interviews and surveys at the Navy's major MTFs, senior Navy 
health care executives told NPS faculty what was important to manage their commands and 
where they saw themselves in terms of competency levels. This information provided the 
bases for designing the 49 education modules. 

The modules are taught by full-time NPS graduate faculty at the Monterey campus 
or at MTF sites. The subject matter within the modules was created as a result of the need 
assessment. These modules are shorter than the courses in the health care management 
programs of several universities, which require one or two years of residency. Additional 
instructional areas that relate exclusively to managing Navy MTFs have been integrated 
into the program. [Ref. 2] 

D. THE EXECUTIVE MANAGEMENT EDUCATION 

The EME Program is designed to prepare medical department officers for executive 
medicine positions. The objectives of this educational program are 1) to refine a manager’s 
administrative viewpoint, which considers the effects of any one decision on the 
organization, 2) to provide a thorough understanding of the precise tools associated with 
operations analysis and economic consequences of any decision, and 3) to illuminate the 
meaning of working in managed care and lead agency environments. These objectives aim 
to provide an executive with increased confidence in coping with both present and future 
military health care management. 
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E. 



STAKEHOLDERS 



A stakeholder is any group that can affect or is affected by the achievement of an 
organization's objectives. [Ref. 3] To be a stakeholder means to have an interest in the 
process or outcome of the issue. Some stakeholders may want things to stay the same; 
some may favor small changes of various sorts; some may want radical change. There may 
be stakeholders who are not involved in any controversy or process, yet have an interest in 
the issue. Active stakeholders are organizations or groups or occasional individuals who 
seek to affect the current initiative because of some interest. Some of them may be public 
spirited or acting on behalf of others. 

The EME initiative has six stakeholders, not all of whom are equally affected by or 
have a significant effect on the initiative. A stakeholder map is illustrated in Figure 1 . 

1 . Commanding Officers and senior military personnel are the most significantly 
affected by this initiative. They are required to complete the EME curriculum in 
order to acquire the necessary proficiency and demonstrate professional 
administrative skills. 

2. Congress is also an important stakeholder because they are the primary decision 
maker who issued a directive to the DoD concerning the required skills, 
knowledge, and experiences an MTF commander must possess. 

3. Clearly, DoD, including the individual military services, has a major stake in 
this EME initiative. The DoD’s position is that the training of senior medical 
personnel is vital for health care executives to achieve the necessary skills. 

4. Professors who are experts in their respective fields comprise another group that 
can exert influence. The obvious impact of this initiative is that, as more 
medical officers go through this program, more professors are needed to teach 
the required skills. 

5. BUMED has a direct impact on the EME initiative. They are involved in the 
design and funding of the EME program based on the competencies identified. 

6. NPS Institute for Defense Education and Analysis (IDEA) is charge with the 
development, delivery and management of EME programs. 
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Commanding Officer & 
Senior Military Personnel 



NPS Institute for Defense Education 
and Analysis (IDEA) 




Bureau of Medicine 
and Surgery 



Department of 
Defense 



Figure 1. A Stakeholder Map 



F. DISTANCE LEARNING 

In today's world, continuous learning has become essential for effective military 
and health care leaders. Technology is quickly strengthening our ability to respond to that 
need, with distance education emerging as a timely and potentially cost-effective way to 
provide educational opportunities. 

Non-traditional education is increasingly identified with distance education or 
education taking place beyond the physical spaces (campuses) traditionally set aside for 
such learning. At its most basic level, distance education takes place when instructor and 
student(s) are separated by physical distance, and technology is used to bridge the gap. 
These types of programs can provide various educational opportunities to students, 
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including executives in both the public and private sectors, particularly those disadvantaged 
by limited time, travel budgets, or physical disability. [Ref. 4] 

Distance education has always benefited from the development of communications. 
There are a number of current distance learning technologies available, such as microwave 
broadcasts of both audio and video, videoteleconferencing, and Network Based Instruction. 

The microwave broadcast classroom normally contains a single camera, which 
concentrates on the instructor, on an experiment, or on the blackboards where notes are 
being written. 

Videoteleconferencing is an interactive distance learning setup using the t-1 phone 
line, which provides two-way audio and video between the broadcast and remote 
classrooms. Major universities and school districts throughout the country are now wired 
for the Integrated Services Digital Network (ISDN), which uses compression technology to 
transmit voice, video, and data simultaneously over standard telephone lines. [Ref. 5] 

The most recent distance learning medium of learning and teaching is Network 
Based Instruction (NBI). This innovative approach for delivering instruction to remote 
audience is accomplished by using the Web. Public and private organizations like DoD, 
colleges and universities; corporations like IBM, Microsoft, and others are investing 
resources in developing these new training modalities. They expect an increase in Web- 
based training within the next few years. [Ref. 5] Organizations, NPS among them, whose 
customers are limited by cost and time constraints will require the flexibility that Web- 
based training can provide. NPS launched three Web-based educational programs in 1997 
and plans additional online programming for EME this year. 

G. NETWORK-BASED INSTRUCTION 

Network-Based Instruction is defined as a hypermedia-based instructional program, 
which utilizes the attributes and resources of the World Wide Web to create a meaningful 
environment where learning is fostered and supported. [Ref. 6] 
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1. Origins 

The Internet began as a project by the Defense Advance Research Project Agency 
(DARPA), whose goal was to develop a highly decentralized network that could both route 
information and survive a nuclear war or natural disasters. The resulting network was 
originally used for sharing information between research laboratories and researchers at 
educational institutions. Eventually, more and more students began to use the network, and 
when companies joined in as well, complete with advertising, the Internet was bom. 

One of the earliest uses of the Internet was for electronic mail. Long before the 
World Wide Web (WWW) and Web pages, the net was used for conducting searches and 
finding and downloading files and programs (FTP). Another early use was the Usenet, used 
by news groups, which was developed in part to deal with the heavy volume of e-mail 
generated by list servers. The hypertext transfer protocol (http) was developed in the 
summer of 1991 at European Particle Physics Laboratory in Geneva, Switzerland as a way 
to share research papers, and the first Web pages were created. [Ref. 7] 

2. Why Network-Based Instruction? 

The use of the WWW is growing exponentially. Estimates range from an increase 
of six percent to one of 20 percent per month. In January of 1991, there were 365.000 
hosts. In January of 1996, there were 9,472,000 hosts. [Ref. 6] In addition to user traffic, 
the creation of the Web servers, home pages, and other digital resources on the Web is 
similarly expanding. This increase represents a tremendous potential for educators. [Ref. 
8 ] 

The WWW and the associated use of hypertext approaches have created new 
opportunities. Over the last year, a number of universities experimented with and started to 
use the Internet to support teaching and administration. Harvard Business School is 
offering executive education via the Web. The use of Web to support part-time MBA was 
seen as high potential, as evidenced by the University of Virginia offering Executive MBA 
for part-time students. [Ref. 9] 
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One success story in providing higher education through NBI is the University of 
Phoenix, a no frills university, with no laboratories, gymnasiums, and dormitories. They 
have 58 campuses in 12 states, Puerto Rico, and dozens of foreign countries. As of 
December 1997, they have an enrollment of 42,000, and are still growing at 20 percent a 
year. This institution provides continuing education, bachelor and master degree programs 
mostly in business and information technology to adults who work full-time and attend 
classes only at night or online. Their online students can complete their university studies 
from virtually anywhere in the world. [Ref. 10] 

NBI is particularly applicable for MBA and executive education courses, just like 
the EME courses for senior DoD medical personnel. Students can enroll and participate in 
these on-line courses by using the Intranet applications. Intranets are internal Web systems 
that work in conjunction with the Internet and WWW. They enable students or employees 
to access school or company information with appropriate passwords. 

The military has adapted to changing technology in providing training and 
education via the Web. One example is the Defense Systems Agency Information Training 
Facility, located in Falls Church, Virginia, which has developed and distributed Information 
Security Training Courses to various Information Security agencies via the Web. [Ref. 1 1] 
The Services accomplish most of their training of individuals in residential school 
settings. For several thousand courses, the Services bear the travel and temporary duty 
(TDY) costs needed to support students in centralized training facilities. This practice 
affects service operations, unit performance, and quality of life for individuals whose work 
schedule and family responsibilities are interrupted by travel to distant training locations. 
These factors have motivated the Services to look for alternatives to centralized training 
facilities. Currently, the need to identify acceptable alternatives like NBI is increasing 
because: (1) budget pressures are reducing funds to operate centralized schools; and (2) with 
modernization and technology enhancements that increase the complexity of military 
equipment and support systems, training demands are increasing. [Ref. 12] 
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The Naval Postgraduate School, through IDEA as the technology leader within DoD 
for adapting new technology to education, started the process of converting EME modules 
to NBI. T his initiative will provide opportunity for anytime, anywhere EME education on 
demand in both synchronous and asynchronous environments. Synchronous instruction 
requires the simultaneous participation of all students and instructors, and the interaction is 
done in real time. Asynchronous instruction does not require the simultaneous participation 
of all students and instructors. They do not need to gather together in the same location at 
the same time. 

As an instructional technology, the Web has the ability to carry a variety of media 
from virtually anywhere, well beyond the traditional means. With access to course 
information and instruction on a flexible schedule through telecommunications, students 
can work at their own pace and to have more time to compose responses. They can learn 
the coursework at home or work, wherever and whenever Internet access is available. [Ref. 
13] 

3. Educational Uses of the Internet 

As indicated earlier, NBI is a form of distance learning in which course modules are 
posted on the WWW. Individuals with access can fog on and connect to the courses at their 
convenience. Presentations on the Internet allow for multiple media types, including text, 
sound, graphics and videos. Finding information on the Internet was made easy through the 
use of hypertext and hypermedia, providing links to a new page with related information. 

The Internet itself is not interactive. It can be used in many ways that vary from 
passive reading, like a library book, to user driven simulations that change direction based 
on choices made by the user, like the video game model. Depending on the instructor's 
time, resources, and imagination, the Web and the rest of the Internet can be used in 
minimally, moderately, or highly interactive ways. Interactivity is a medium's ability for 
senders and receivers to notice and respond to each other’s communication cues. 
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a. Minimally Interactive Use of the Internet 

Minimally interactive use of Internet is the lowest level of interactivity. 
There is a limited or no interaction between the student, instructor, or expert at the distant 
locations. Listed below are several examples of minimally interactive uses. 

(1) Research Tools. The Web offers many kinds of research tools. 
Search engines, government data sites, sites by subject-matter experts, and professional 
resources are just as few. Examples are sites by National Aeronautics Administration, 
United States Geological Survey, and other agencies. 

(2) Syllabus Assignment Listings. This is the simplest and least 
interactive use of the course Web page. Provided are a copy of the course syllabus, 
assignments, contact information, perhaps a photograph of the professor and a short 
biography, if desired. The syllabus can be updated as often as necessary to keep it current. 
A list of discussion questions could be added so those students could keep them in mind 
while reviewing the last class or preparing for the next. This kind of page is fairly easy to 
create and maintain with minimum effort. 

b. Moderately Interactive Use of the Internet 

Moderately interactive use of Internet has a conservative degree of 
interactivity between the student, instructor, or expert at the distant locations. Instructors 
and experts may act as facilitators; and they can provide support, feedback, and guidance 
mostly via asynchronous communication, like e-mail, which allows for time-independent 
interaction. 

(1) Newsgroup-like Discussion Groups. Discussion groups can be 
set up so that the professor can post questions for the students to discuss via e-mail. Each 
response is sent to a server which publishes the message in a list for other students to view. 
These groups can be made private so that only students enrolled in the course has access to 
the lists. If large classes make in-class discussions impractical or if anonymous discussion 
is desired, then a Web-based group discussion may be a useful addition to a course Web 
site. 
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(2) Online Chat Sessions. These sessions occur in real time, which 
means that as long as at least two people are logged in, a live discussion can take place. 
When one participant types a message, all other participants see the message at once. These 
sessions can be embedded directly in a Web page or accessed with special software 
designed for the purpose. 

(3) Quizzes, Surveys and Appointment Books. Forms can be added 
to a Web site with a little effort. An example of a form is a quiz that each student fills out 
and submits electronically. It can be e-mailed to the professor, automatically graded, and 
added to the database of that student’s work. 

c. Highly Interactive Uses of Internet 

Highly interactivity use of the Internet occurs when NBI students can 
interact with each other, with instructors, and on-line resources in real time. Instructors and 
experts may act as facilitators. They can provide support, feedback, and guidance via 
synchronous communication, like conferencing tools, which allows for live interaction. 

(1) Real Time Interaction With Others. Real time interaction can be 
done on the Web in several ways. One is to have a link that opens another application, 
which makes a connection with a server. Two types of real time interaction, which are 
accessed this way, are Internet Relay Chat (IRC) and Multi User Object Oriented (MOO). 
IRC and MOO allow virtual gathering of students and teachers from around the globe at a 
set time to discuss a set of topics. 

As technical advances make resources more accessible, the Web will 
be a viable medium to facilitate learning. NBI has the ability to provide a learning 
environment that is global and interactive. This technology can make it easier and quicker 
for organizations to train and educate their personnel anytime and anywhere. [Ref. 6] 

4. Learning Styles 

Distance learning may extend access to learning opportunities to students who 
otherwise might not be served. However, options that merely replicate the problems and 
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failures of the conventional classroom will not benefit students. To create options that 
enhance learning, we must consider different learning styles in the context of distance 
learning. 

A Web-based environment can accommodate flexibility in learning styles. The way 
an individual learner reacts to the overall learning environment makes up his or her learning 
style. No universally accepted terminology exists to describe learning style and its various 
components; however, that people react to their learning environment is a core concept. 
James and Blank (1993) define learning style as the complex manner in which learners most 
efficiently and effectively perceive, process, store, and recall what they are attempting to 
learn. [Ref. 14] Learning style has three distinct but interconnected dimensions: the 
perceptual (physiological or sensory) mode, the cognitive (mental or information 
processing) mode, and the affective (emotional or personality characteristic) mode. [Ref. 5] 

a. Perceptual dimension 

Perceptual dimension identifies the way individuals assimilate information. 
It involves the body's biological response to external stimuli, including input through 
physiological factors such as speech, movement and any of the five senses. This dimension 
depends on the physical ability of an individual’s body to integrate information into the 
brain. It is the means through which the information is extracted from the environment 

b. Cognitive dimension 

The cognitive dimension includes the storage and retrieval of information in 
the brain. Information-processing habits represent the learner’s typical way of perceiving, 
thinking, problem solving, and remembering. Each learner has preferred avenues of 
perception, organization, and retention that are distinctive and consistent. 

c. Affective dimension 

Affective dimension encompasses aspects of personality that relate to 
attention, emotion, and valuing. Affective learning styles are the learner’s typical mode of 
arousing, directing, and sustaining behavior. Although the components of the affective 
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learning style cannot be observed directly, they can be inferred from the learner's behavior 
and interaction with the environment. 

For the instructor developing a Web-based course, the challenge is to 
analyze the learning style characteristics of the group and develop modes of instruction that 
fit those characteristics. Unfortunately, instructors are only able to identify a few elements 
of their students' learning styles through observation. Consequently, instructors should 
adapt instruction to the variety of ways individuals learn by developing and planning 
various tasks that cater to various learning styles. 

Learning grows out of what interests the learner. The Web offers an 
excellent environment to support a number of learning style differences. Faculty can create 
a learning environment that challenges each style of student to pursue a topic further and 
gather different perspectives on issues. [Ref. 5] 

5. Cost Effectiveness 

The conversion of EME modules from traditional classrooms to NBI requires a 
large investment. Conversion requires an investment up front while resident training 
programs continue; payback, in the form of travel and per diem cost savings, begins as NBI 
comes online and replaces resident courses. Such an investment is difficult to defend 
without considering the costs and benefits of converting EME modules to NBI. 

Two of the greatest potential benefits are increased training effectiveness and the 
saving of travel and per diem costs for resident training. Training effectiveness is a 
complex outcome measure. It includes the ability of NBI to provide at least the same 
amount of knowledge and as well-educated of a learner compared to the traditional 
classroom, quicker ability to update course materials, and increase throughput of the 
number of students. [Ref. 12] 

A Naval officer thesis study on Cost Effectiveness Analysis of Converting a 
Classroom Course to a Network Based Instruction determined that converting only two 
EME modules to NBI was not cost-effective. [Ref. 1 5] The conversion of more modules, 
however, could generate cost savings. 
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Modules presented during the first week of EME conference were of varying length 
with some modules lasting for two, three or four hours. On the second and the third weeks 
of the EME conference, each day was broken into two, 4-hour course modules with one 
module in the morning and the second in the afternoon. The study was based on the use of 
two, 4-hour modules per day. If two modules are converted, one fewer EME conference 
day is required and its costs are eliminated. The cost per student to present one EME 
conference is $13, 348.37. The total cost per student actually increases for each module 
converted to NBL until four EME conferences have been held. Only when 20 modules are 
converted to NBI, and four EME conferences have been held will the cost per student for an 
EME conference with converted modules become lower than the cost per student for an 
EME conference without converted modules. However, the savings are minor at $142.22 
per student. Obviously as the number of converted module increases, the conference length 
in days decreases, thus decreasing the conference room costs, opportunity cost of student 
salaries, and per diem costs. There is a greater saving after the conversion of the 20th 
module, since weekend travel costs are eliminated. The savings continue to increase for 
each even number of modules that are converted. If all modules are converted, a savings of 
$1,234.12 per student is realized. [Ref. 15] 

Stanford Technologies Group, one of the contractors currently transforming an 
EME module to NBI, conducted an Economic Analysis of Training Alternatives consisting 
of three cases: (1) on-the job with the instructor traveling to the student and with no student 
travel; (2) NBI provided to the student, no travel involved; and (3) centralized classroom, 
both students and instructors travel. Based on their study, the total cost of training strategy 
case number two, based on 1,000 students, is $270,000, or $270.00 per student. NBI 
instruction is $143,600.00 less expensive compared to case one and $280,000.87 less 
expensive compared to case three. Figure 2 shows the results of the economic analysis of 
training alternatives. 



16 



Economic Analysis of Training Alternatives 

Notes for using this worksheet 

(a) Update the grey fields as needed for each business case 



(b) Number of tnps MUST be manually updated 

(c) Travel cost per instructor should consider the possibility of 
multiple training sessions, back-to-back, at one training site 

(d) CBT usually teaches the same content in 50-75% of the time 
required for classroom, instructor-led training 

(e) Do not overlook packaging and shipping costs for student 
training matenals 



Case #1 


Case #2 


Case #3 


On-the-job 
with instructor 
travelling to 
the student 
No student 
travel 


IBT provided 
to the student. 
No travel 
involved 


Centralized 
classroom 
Both students 
and instructor 
may travel. 



Student-Related Expenses 


Number of Students 


1000 


1000 


1000 


# Students per Class 


4 


1 


12 


# Training Sessions Needed 


250 


1000 


83 33 


Length of Training Session (in hours) 


8 


5 


8 


Travel Time to/from Training per Student {in hours) 


0 


0 


6 


Student Salary per Hour 


$20.00 


$20 00 


$20 00 


Total Cost of Student Salanes 


$160,000 00 


$100.000 00 


$280,000 00 


# of Tnps for Students to the Training Site (total) 


0 


0 


1,000 


Travel Cost per Student (including travel, food & lodging) 


$0 00 


$0 00 


$50 00 


Total Cost of Student Travel 


$0 00 


$0 00 


$50,000 00 


Total Student-Related Expenses 


$160,000 00 


$100,000 00 


$330,000 00 



Instructor-Related Expenses 


# Training Sessions 


250 


1000 


83 33 


Length of Training Session (in hours) 


8 


5 


8 


# Instructors per Training Session 


1 


0 


2 


Initial Instructor Prep Time (per instructor, in hours, not me 
matenal 


40 


0 


40 


Instructor Preparatory Time per Session (per instructor, in hours) 


2 


0 


4 


Instructor Salary per Hour 


$40 00 


$40 00 


$40 00 


Travel Time to/from Training Site per Instructor (in hours) 


8 


0 


8 


Total Cost of Instructor Salary 


$181,600 00 


$0 00 


$136,533 33 


n of Tnps per Instructor to the Training Sites (total) 


100 


100 


40 


Travel Cost per Tnp per Instructor (including travel, food & 
lodging 


$500 00 


$0 00 


$200 00 


Total Cost of Instructor Travel 


$50,000 00 


$0 00 


$16,000 00 


Total Instructor-Related Expenses 


$231,600 00 




$152,533 33 



Other Related Expenses 


n Training Sessions Needed 


250 00 


1000 00 


83 33 


Cost of Classroom per Session 


$0 00 


$0 00 


$250 00 


Total Cost of Classrooms 


$0 00 


$0 00 


$20,833 33 


Cost of Equipment per Student 


$0.00 


$0 00 


$25 00 


# Students 


1000 


1000 


1000 


Total Cost of Equipment 


0 


0 


25000 


Sunk Cost to Develop Student Matenals (Workbooks. CBT. etc ) 


$10,000.00 


$150,000 00 


$10,000.00 


Cost per Student for Matenals (Copies. Replication. Shipping. 

etc.) 


$1200 


$20.00 


$1200 


Total Cost of Student Matenals 


$22,000 00 


$170,000 00 


$22.000 00 


Total Other Related Expenses 


$22.000 00 


$170,000 00 


S67.833.33 



Summary of Costs 


Total Student-Related Expenses 


$160,000.00 


$100.000 00 


$330,000 00 


Total Instructor-Related Expenses 


$231.600 00 


$0 00 


$152,533 33 


Total Other Related Expenses 


$22,000 00 


$170,000 00 


$67,833.33 


Total Cost of Training Strategy 


$413.600 00 


$270,000 00 


$550,366 67 


# Students 


1000 


1000 


1000 


Cost per Student 


$413.60 


$270 00 


$550 37 



Figure 2. Economic Analysis of Training Alternatives 

Although the sample analysis conducted by Stanford is completely constructive, just 
bear in mind that the data used were only approximate. One example is the data they used 
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for instructor's salary, which is $40.00 per hour, while the instructor's at NPS may be as 
high as $300.00 per hour. 1'he $270,000 total training cost for case two is really coming 
from the sunk cost of $150,000 that is used to develop student course materials. The 
question is, What are they getting with this $150,000? If they will get an engaging, robust 
distributive learning module, then the answer may be yes. However, if they will get a 
totally asynchronous text page turning module, even when the dollar value per student is 
down that may not be the correct pedagogy. Everything must be taken into consideration 
when evaluating alternatives. If they are just looking to reduce cost as their bottom line, 
then they will look for the cheapest way to get the learning out to the students. The 
resulting product may not be good, their credibility goes down, and learners are not getting 
the required education. However, if they are looking for the quality of education, they may 
be willing to spend the same amount of money per student and be able to provide to twice 
as much the number of students. 

H. SUMMARY 

As the Information Age evolves and technical advances make resources more 
accessible, NBI is becoming a more viable medium to facilitate learning. Several 
organizations, including the DoD, schools, and private companies have adapted to this 
changing technology. NBI make it easier and quicker for organizations to train and educate 
their personnel anytime and anywhere. However, conversion of EME modules to NBI 
requires a large investment up-front. Conversion of more modules will reduce the number 
of training days, resulting in greater savings. The potential benefits are increased education 
or training effectiveness and savings of travel and per diem costs for resident training. 
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HI. DECISION PROCESSES IN THE CONVERSION 



A. INTRODUCTION 

I his chapter describes the initial decision process used in the conversion of an EME 
module to NBI. The focus will be on the criteria used by IDEA in the selection of modules, 
instructors, and contractors. 

This pilot project, the first time that an EME module will be converted to NBI, 
seeks to improve the current process so that future conversion of modules will be more 
effective. 

1. Background 

The Naval School of Health and Sciences (NSHS) is a BUMED command in charge 
of the management of the EME Program. The Institute for Defense Education and Analysis 
(IDEA), a department within the office of the Associate Provost for Innovation, is the 
designated coordinator of the EME Program. 

On April 30, 1997, IDEA submitted a proposal to study, implement, and evaluate 
distance learning applications for EME to the Commanding Officer, NSHS. This proposal 
included empirical research, modular conversions, measure of effectiveness and delivery of 
the plan. [Ref. 16] When it submitted this proposal, IDEA had no established criteria for 
the selection of modules, instructors, and vendors for the conversion to NBI. 

2. Institute for Defense Education and Analysis (IDEA) 

IDEA is charged with developing strategic partnerships with select universities, 
government agencies, and private organizations to foster a mutually supportive environment 
for innovative collaboration in the design, creation, and implementation of an effective 
distributed learning systems. 
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a. Mission 

IDEA'S mission is to identify, develop, market, and manage innovative 
programs of applied research, consultation, and teaching, including non-residential 
executive and continuing programs, on a cost-reimbursable basis, in support of the DoD 
requirements. 

IDEA'S educational programs are designed to conform with the NPS 
resident Graduate Curricula. They contribute to NPS faculty/student development, 
instruction, and research and build upon the NPS core mission. The NPS mission is to 
increase the combat effectiveness of United States and Allied armed forces and enhance the 
security of the United States of America through advanced education and research programs 
focused on the technical, analytical, and managerial tools needed to confront defense-related 
challenges. IDEA’S programs contribute to educational and training innovation, including 
distance/distributed learning and non-traditional educational methods. 

b. Organization 

The Associate Provost for Innovation serves as the Director of IDEA. 
Currently, three organizational subdivisions and two centers support IDEA’S mission: 
Education, Applied Research, and Consultative Services; and The Center for Acquisition 
Education, Training, and Research and the Naval Center for Acquisition for Education. 
Figure 3 shows IDEA’S organizational structure. Its staff consists of NPS professors and 
civilian professionals who are experts in their fields. Currently, IDEA has limited 
programming capability, and has no immediate plans to establish an in-house development 
capability. 

c. Core Competencies 

IDEA’s core competencies include: (1) development and delivery of 
executive education programs; (2) distributed learning /network-based instruction course 
conversion and development; (3) consulting and applied research program development 
and support; (4) program and project management; (5) conference coordination and 
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logistics support: and (6) survey development and execution, both automated and traditional 
solutions. 



Office of the Associate Provost for Innovation 




Figure 3. IDEA'S Organizational Structure 
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B. 



NBI MODULE SELECTION PROCESS 



I he selection process for the selection of modules, instructors, and vendors starts 
with the Distributive Learning Team. The team is made up of four to five members, who 
are faculty and staff members of NPS. Their responsibility includes: (1) oversee and 
manage the delivery of residential and distributed graduate education. (2) create or adapt 
content to new distributed learning pedagogies, (3) explore new instructional system 
technologies to foster collaborative learning environments (synchronous and 
asynchronous), (3) provide analysis in determining course/class attributes, (4) facilitate 
customer support/interface between facilities, contractors, and subject matter experts, and 
(5) conduct cost-benefit analysis of alternatives. The team will identify and evaluate the 
soundness of alternative choices for module, instructor and contractor selection, and submit 
their recommendation to the Director and Executive Director of IDEA for the final decision. 
[Ref. 17] 

1. Proposal Development 

The initial decision process to convert current EME modules to NBI started when 
IDEA submitted its proposal to NSHS. This proposal laid out the deliverables, which 
focused on two distance-learning alternatives: video teleducation (VTE) and NBI 
interactive remote learning. Also explained were the contents, methodology, strategy and 
budget requirements for these deliverables. [Ref. 16] 

The proposal to convert EME modules to NBI was approved and funded by 
BUMED for $2.5 million. This funding includes traditional classroom instruction and 
conversion of modules to NBI and VTE. [Ref. 18] The Distributive Learning Team, 
formed to jump-start this initiative, was given the task of selecting modules, instructors, and 
vendors to start the conversion process. 

2. Module selection 

Different EME modules have different module content. The two modules selected 
for the pilot study were Module 24 (Budgeting for Defense Health) and Module 51 
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(Medical Readiness). According to the Executive Director of IDEA. “Those two modules 
were selected because they are tightly developed, well written and organized. Also, the 
modules lasted only for a short period of time, no more than four hours.” [Ref. 17] The 
EME Program Coordinator of IDEA said that, “One of the criteria that was used was the 
study conducted by Crawford and Suchan, “Media Selection in Graduate Education.” [Ref. 
19] Although NBI was not included in Crawford and Suchan's media selection study, 
according to the author. VTE’s use of 50 percent or more on Know and Supply Information 
and Apply Information to Structured Process Learning Outcomes can be apply to NBI’s. 
Also, according to the survey Crawford and Suchan conducted, those two modules do not 
require interpersonally complex interaction to attain their learning outcome, which made 
them excellent candidates for conversion to NBI. [Ref. 20] 

Many see the WWW as the next great alternative to traditional instruction. 
According to Romiszowski, “Given recent developments in telecommunications 
technology, NBI can replicate virtually, all the key learning activities that occur in 
traditional classroom-based courses.” [Ref. 21] Although EME modules can be converted 
to NBI, the question is how well can they be converted? Those that require a significant 
amount of personal live interaction may be extremely difficult to transform to NBI. While 
there are limitations, as with other distance education technologies in the degree to which it 
can replicate traditional instructional events, NBI can be an enhancement over traditional 
classroom-based and distance education environments. 

3. Instructor Selection 

Selection of the most qualified professor is very important for the success of this 
pilot program since he or she will provide the course content of the module. Although there 
are many excellent professors at NPS, Professor Doyle was selected to convert his modules 
to NBI based on his reputation and accomplishments. He was the 1996 recipient of the 
prestigious RADM John J. Schieffelin Award for excellence in teaching. Well respected by 
students, staff, and faculty, he is also motivated and open to new instructional design ideas. 
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In addition, he is computer literate, especially comfortable with the use of Power Point for 
presentations. [Ref. 19] 

4. Contractor Selection 

To design Web-based courses, design teams are needed. The purpose of the design 
team is to unite a strong variety of experiences and perspectives, with the educational 
learning outcomes as the central impetus and innovative technologies such as NBI as the 
medium. The objective of the design team is to create educational content that is of the 
highest caliber and to deliver it using the most relevant technologies available. The design 
team is comprised of discipline-specific experts. Web-design professionals, program 
administrators, standards-based education consultants, and educational technologists. 
Currently, IDEA does not have a Web design team to transform modules to NBI. In order 
to accomplish this conversion, they needed to contract for a Web design team. 

The Executive Director of IDEA stated, “We do not have an established criteria or 
decision process in the selection for contractor or vendor.” This will be the first time that an 
EME course module will be transformed to NBI at NPS. The management of IDEA made 
the decision to convert two EME modules to NBI, with funding authorization from 
BUMED. One of their purposes for conducting this pilot study is to learn and establish 
criteria for selection of contractors, modules, instructors, and other requirements. Their 
goal is to learn how to effectively coordinate different NBI components and processes, 
which include the following: (1) content development, (2) multimedia components, (3) 
Internet tools, (4) computers and storage devices, (5) connections and service providers, (6) 
authoring programs, (7) servers, and (8) browsers and other applications. [Ref. 17] 

IDEA started the contractor selection process by contacting approximately a dozen 
contractors to make a presentation or demonstration of their products. There were eight 
contractors who made presentations. Two contractors were selected, Stanford Technologies 
Group and Arista Systems, Incorporated, because they were able to demonstrate to IDEA 
management’s satisfaction that they were capable of converting EME modules to NBI. 
[Ref. 17] 
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Stanford Technologies Group (STG), based in Washington D.C., was selected 
because of their prior military training development experience. However, because they do 
not have experience in developing courseware products, they established a partnership with 
Electronic Learning Facilitators (ELF), Inc. to provide computer services including 
analysis, design, and courseware for NBI modules to win the contract. The company 
demonstrated a NBI training module of how to play a woodwind instrument and conducted 
an Economic Analysis of Training Alternatives. Based on this work, they were assigned to 
do the conversion of Module-24, Budgeting for Defense Health. There was no specific 
reason why IDEA selected STG to transform Module-24; according to the Executive 
Director, IDEA, “We just want to assign a module to each vendor.” [Ref. 17] 

Arista Systems, Inc., the other contractor selected, is based in Monterey County. It 
has partnerships and alliances with California State University Monterey Bay, The 
Laurasian Institution, and Nousoft, Inc. Arista has experience with distance technology 
through its partnership with Lauratian Institution. To win the contract, they were able to 
show a course module via the Internet, titled “Doing Business in Japan.” That was one of 
several modules that they had developed for the Asian Pacific Economic Countries 
sponsored by the Virtual University of Asia. Arista is also putting together an Accredix 
Platform Neutral Course Management System, which supports the delivery of instructional 
material and includes the management of network-based environment. Modules that are 
converted by other contractors can be used within this system. Another reason for Arista's 
selection was its Monterey location, making the company more accessible during the 
conversion. Arista was assigned to do Module 51, Medical Readiness. 

C. SUMMARY 

IDEA did not use a structured decision process or specific criteria to select modules, 
instructors, and contractors in the conversion of an EME module to NBI. Their goal was to 
learn from this pilot study and to establish criteria and a process for future conversion 
projects. 
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The next chapter will offer recommended criteria on the selection process and assess 
the decision process used by IDEA compared to other organizations. 



t 
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IV. ASSESSMENT OF THE DECISION PROCESSES 



A. INTRODUCTION 

This chapter assesses the decision process used by IDEA in transforming an EME 
module to NBI and provides recommended criteria for the selection of module, instructor 
and vendor. It will also discuss processes used by other organizations in their conversion 
projects to serve as a model for the conversion process and provide insight on how to plan 
the conversion process. 

B. THE DECISION PROCESS 

The decision to develop and maintain a new instructional system, such as the 
conversion of EME modules to NBI, is a large undertaking. This undertaking should be 
based on a reasonable set of probable outcomes and/or benefits. IDEA management and the 
Distributive Learning Team, as the managers of this project, are entrusted with the 
challenge of planning for this kind of change. The new instructional system they 
implement must ensure that learners develop the new knowledge, skills, and attitudes 
required to meet program objectives. To meet program objectives, IDEA management and 
team members should use models such as the Instructional Systems Development Model to 
plan, develop, and deliver new systems to deliver educational and training programs. 

1. Instructional Systems Development 

Instructional Systems Development (ISD) procedures have had an impact on the 
way education and training programs are planned, developed and delivered. ISD has 
military beginnings. The landmark event in ISD was the Inter-service Procedures for 
Instructional Systems Development Project in 1975, which offered a model for using ISD in 
the military. The ISD model also influenced the development of educational and training 
programs in business and industrial settings. [Ref. 22] Although the ISD model is used 
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generally to systematically develop training programs, this model can be used profitably to 
develop new instructional technologies. 

Principles of ISD are more applicable when the expected outcomes of the education 
or training can be clearly established, and the organization providing the education is held 
accountable for producing results. ISD projects flourish in organizations that place an 
economic value on the time of its learners. 

ISD is not a remedy for all education and training problems; nor is it guaranteed to 
be equally effective in all situations. The purpose of ISD is to assist in fulfilling the 
requirements for the successful implementation of educational and training programs. 

All ISD models have several things in common. First, they are process-oriented and 
not product-oriented. All too often, programs are implemented without a thorough analysis. 
This model will help the decision-makers to make the right decision in developing and 
managing educational programs. Second, the ISD approach is systematic and deliberately 
planned. Third, all ISD models have five stages, each with a similar name to describe a 
similar purpose. Fourth, output from one phase acts as input for the next. Finally, this 
process has built-in controls for review and required revision. 

The intent of this ISD model is to present information about the process one goes 
through in developing instructional systems, rather than a fixed, step-by-step sequence of 
specific actions. This model will be used to assess the decision process used by IDEA and 
other organizations in the conversion of their courses and modules to NBI. The five stages 
of the ISD model are: 

a. Analysis Phase 

The process begins with the Analysis Phase, in which the educational or 
training needs of the different groups of students within the organization are analyzed. This 
phase, one of the most complex and time-consuming, begins with few or no assumptions 
about what education or training is required. This phase defines the problem, identifies the 
source of the problem, and determines possible solutions. The phase may include specific 
research techniques, such as need analysis, job analysis, and task analysis. The outputs of 
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this phase otten include the instructional goals and a list of tasks to be taught. These 
outputs will serv e as the inputs for the design phase. 

b. Design Phase 

The Design phase involves using the outputs from the analysis phase to plan 
the strategy for developing the instruction. This phase develops an outline for reaching the 
instructional goals determined during the analysis phase and expands on the instructional 
foundation. Objectives are developed and used to determine appropriate content and 
methods. Some of the elements of this phase may include writing a target population 
description, conducting a learning analysis, writing objectives and test items, and 
sequencing the instruction. The outputs of the Design phase will be inputs for the 
Development phase. 

c. Development Phase 

The Development phase builds on both the Analysis and the Design phases. 
The purpose of this phase is to generate the lesson plans and materials. This phase involves 
the development of the educational materials, which include the content the instructor will 
present, as well as other supporting materials. Designers often work closely with functional 
specialists who review and even write the first draft of the materials. 

d. Implementation Phase 

The Implementation phase refers to the actual delivery of the instruction, 
whether it is classroom-room based, lab-based, or computer-based. The purpose of this 
phase is the effective and efficient delivery of instruction. It generally begins with a pilot 
module, in which materials and methods are formatively evaluated before they are made 
ready for mass use. This phase must promote the students’ understanding of material, 
support the students’ mastery of objectives, and ensure the transfer of knowledge from the 
instructional setting to the job. 
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Evaluation Phase 



e. 

The tinal evaluation phase assesses the training at the end of each 
presentation to determine whether the students have mastered the material and whether the 
instruction itself was appropriate. Evaluation should actually occur throughout the entire 
instructional design process, within phases, between phases, and after implementation. A 
second type of evaluation, which generally occurs some months after the training, is 
designed to follow up on the students' use of what they learned in order to determine 
overall program success. 

The purpose of this five-stage process is to present a model that should be 
used and adjusted according to the users' unique environment and needs. Not everybody 
will use it the same way. Some may perform certain tasks simultaneously, or in less detail, 
or not at all. 

This ISD model can be applied in the decision process to convert EME 
modules to NBI. This model can provide procedural frameworks for systematically making 
new instructional systems decisions. Part of this ISD process was used to determine the 
learning goals and content of the EME to be converted. 

C. CREATING AN INITIAL DECISION MAKING PROCESS FOR NBI 

Although there are five different phases in the ISD model, the scope of this study is 
the initial decision process, which occurs mainly in the Analysis phase. Figure 4 is a 
proposed initial decision process using the Analysis Phase of the ISD Model for conversion 
of EME modules to NBI. The five step process include: (1) identify the instructional goals; 
(2) conduct needs assessments; (3) establish criteria for module, instructor, and vendor 
selection; (4) select delivery systems, module, instructor, and vendor; and (5) plan the 
project. [Ref. 22] 

The first step is the identification of goals. Usually, this is in response to 
educational and training needs. However, in this case, the goal is to convert traditional 
classroom EME modules to NBI. 
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Figure 4. Proposed Initial Decision Process 



The next step, conducting needs assessments, involves the identification, evaluation, 
and prioritization of educational and training needs to allow the development of the course. 
Part of this approach, the ability of the individuals or the team to meet performance 
expectations for a given job or task, is analyzed. It is important to emphasize that no two 
needs assessment are alike. Decisions and plans will vary with the nature of the goals and 
objectives of the organization. 
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The third step is establishing criteria for module, instructor, and vendor. One 
challenge for instructional designers is the task of suggesting the best criteria for designing 
NBI course modules. Presently, no "best criteria" for the design of NBI exist. The 
selection criteria for module, instructor, and vendor suggested here are based on the 
literature describing other organizations' processes. 

In module selection, various factors must be considered in relation to the medium of 
instruction to be used, fhese include: type of content, number and location of students, 
time available, instructional budget, and facilities available. Matching module requirements 
to technology is a challenging task because of the many variables involved. 

There are several commercial tools available to aid in module selection. One of 
these, the Advisor 2.0, is widely used and is a candidate for standardizing the process of 
media selection by the U.S. Air Force Education and Training Command. [Ref. 23] Also, 
this is one of the two media selection tools being considered by Chief of Naval Education 
and Training Command and is now being used by some Navy Training Commands, such as 
Fleet Training Center, San Diego. [Ref. 24] Dr. J. Bahlis, the author of this program, 
worked with the Air Force to produce a program that meets or exceeds Air Force 
requirements. [Ref. 25] Used in the support of the recommended process, Advisor is not 
only useful in media selection, but also helps organizations analyze costs, impact, and other 
issues associated with evaluating and choosing training. The Appendix is a summary of 
inputs used to generate a sample session created by the author. Though the sample course 
“Module 24” (Budgeting for Defense Health) is completely constructive, the corresponding 
EME course was kept in mind as the author followed Advisor through the step-by-step 
assessment of educational needs. What is immediately obvious is the programs format is 
conducive to using it as a structured interview tool. Bahlis groups the “interview” questions 
into four general areas, each covering a different aspect of what he calls “Factors that 
Impact Media Selection.” 

The “Regulations” section covers the policies and procedures that are fundamental 
to the course and course environment: Why does the course exist? Is the course subject 
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matter or material classified? What type of testing is required? The sample course was 
designed as a public domain course which would require the student to demonstrate skills 
and knowledge during testing. 

The ''Organizational" section helps describe the administrative environment of the 
course: How much time is available to develop the course? What assets are available to do 
so? It also extracts information about the course objectives, contents, and status. 
Consistency in delivery was important and a flexible delivery schedule was required. 

The ’Training" section of the interview investigates the students and the instructors. 
What are their backgrounds? Where are they located? Is it essential or just desirable for 
them to interact? Is the success of the course dependent on their ability to share experiences 
with one another. 

The final part of the interview, “Instructional Needs,” tries to capture what type of 
training environment can be expected: Are the students computer literate? Do they have 
access to PC’s? The Internet? Can they travel to a VTT classroom or CBT center? 

When the interview is complete, Advisor compares the requirements and matches 
them to currently available educational techniques. Advisor produces an “effectiveness” 
score, with 100 percent being an ideal choice. On demand. Advisor produces a list of 
reasons why each candidate technology is less than ideally suited for a particular course. In 
the sample case for example, Intemet/Intranet scores 83 percent because of some deleterious 
environmental factors like student resistance to change. Understanding the logic underlying 
Advisor enables the user to manipulate the variables to generate alternative paths. In the 
sample, a print-based was rejected outright because one of the parameters selected was a 
requirement for audio. If the requirement for audio is negotiable, a correspondence solution 
path can be made feasible, with an effectiveness of 60 percent, by changing the value of the 
variable audio from “required” to “desired.” 

As discussed in Chapter III, the study conducted by Crawford and Suchan on 
“Media Selection in Graduate Education for Navy Medical Officers” demonstrates an 
approach for systematically selecting instructional media for specific learning applications 
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that places priority on the desired learning outcome and the media required to support the 
instructional techniques to attain that outcome. On this study, the media selection 
guidelines were used to match each module with VTE, digitized video disc (DVD), or 
traditional classroom instruction. [Ref. 20] This study can also be useful in conjunction 
with Advisor 2.0. 

The fourth step is the establishment of selection criteria for the instructor. A critical 
part of the EME module conversion to NBI is the instructor. With the current technology, 
NBI can be designed so that the instructor plays the role of facilitator, coach, mentor, and 
guide. Instructors are guided by Web Design Team in specific methods and strategies for 
facilitating courses online. 

To successfully facilitate an online course module, instructors must meet the 
following professional criteria: (1) appropriate content knowledge and experience; (2) 
competence with technologies required to deliver the course successfully— i.e., e-mail, real- 
time communications, group mailing lists; (3) ability to introduce and manage both 
synchronous and asynchronous communications and interactions; and (4) openness to 
emerging technologies and ways to apply them to online educational environments. [Ref. 
26] 

Vendor selection criteria are especially important for organizations that outsource 
their Web-Design Team. Most distance education programs in higher learning centers, such 
as NPS, do not necessarily have all the expertise to perform design and production tasks. 
The expertise lacking is most often in Web programming, multimedia authoring, and media 
conversion skills. A design team is needed to fulfill those requirements. Such a team is 
involved in the design and development of a Web-based module with the objectives of 
creating the highest-caliber educational content and delivering it with the most relevant 
technologies available. For some organizations, it is financially prudent to contract their 
conversion project out to qualified vendors. 

To achieve the milestones of a Web-based initiative, the following requirements and 
criteria for vendor selection are suggested: (1) vendor quality as evidenced by work 
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samples; (2) vendor's price; (3) vendor's understanding of the subject matter; (4) vendor's ' 
experience in the actual conversion of course modules to NBI; (5) quality of the proposed 
solution; and (6) vendor's ' reliability and track record. [Ref. 27] 

I his fourth step, the selection of module, instructor, and vendors, must be based on 
the management and Distributive Learning Team’s approved, established criteria. 

Planning the start of the project is the fifth step, including establishing the timelines 
and resolving other issues based on the information generated from the above steps. 

D. ASSESSMENT OF OTHER ORGANIZATIONS' PROCESSES 

An assessment of the initial decision process used by two organizations, the 
University of Minnesota and Jefferson County Public School, will be discussed using the 
Initial Phase of the ISD Model. Its purpose is to point out the similarities and differences in 
the steps, issues, and constraints involved in their decision process as compared to IDEA’S 
decision process. 

1. University of Minnesota 

The following is the assessment of the initial decision process used by the 
University of Minnesota in designing and implementing college-wide Web-Based courses: 

a. Identify Goals 

In fall 1996, the University of Minnesota’s College of Education and Human 
Development (CEHD) administration asked the Web Development Team to design, 
develop, and implement a traditional college course for Web-Based Instruction. Their 
purpose is to design and maintain a standardized WWW presence for the college 
representing communications, student services, and instructional uses of the Web. Also, to 
create interest among instructors to develop and manage their own Web-Based course 
materials. [Ref. 28] 
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b. 



Needs assessment 



rhe management and the present Web-Based Design Team started by 
assessing whether they had the required expertise and knowledge for the conversion 
process. Based on their assessment, they hired additional members for the Web 
Development and Design team to jump-start the initiative. Also included in the team were 
several members of the communication staff, a representative from the Technical and 
Information Services unit of the college, representatives from student support services, and 
the Assistant Dean of the college. 

c. Establish Criteria 

The University of Minnesota does not have an establish criteria in the 
courses and instructor selection, and vendor selection criteria were not applicable 

d. Selecting module , instructor, and vendor 

The team selected Course number 5145 (Social Studies and the Internet), 
which is mainly a lecture course. Also, the instructor selected was knowledgeable with the 
use of computers and motivated to work with the team in the conversion process. Selecting 
vendor is not applicable to this example since they have their own Web Development and 
Design Team. 

e. Planning the Project 

In planning their project, the Design Team’s tasks were broken into three 
main areas: the Internet, the Intranet, and instructional uses of technology. The Internet was 
their first priority, as the existing site had been designed and implemented four years before. 
The initial redesign of the Intranet followed, with page graphics and templates designed to 
complement the college Internet site. [Ref. 28] 

f Other Issues and Constraints 

The following constraints and issues were also identified in addition to the 
above assessment: 
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1. Lack of Funds and Bureaucracy: The management rejected some of the early 
ideas of technical experts. While the design team was busy resolving some of 
the hardware problems, the conversion process was hampered by the lack of 
funds, end-user hesitation to switch platforms, and the bureaucracy inherent in 
any large organization. 

2. Instructors' Opposition: Some instructors voiced opposition to the plan of 
individual professors converting their own courses to Web-Based Instruction. 
In any group of users, knowledge on a particular subject will vary. While some 
of the instructors are well versed in effectively using the current technologies, 
while many are poorly prepared or have no knowledge of the latest advances. 
The Web-Based Design Team started the process of educating the faculty in the 
area of Internet development. They created the World Wide Web toolbox with 
the intention of providing a repository of information relating to the 
development and use of Web pages in the college. The toolbox was well 
received, but failed to generate significant interest from the faculty in Web- 
based courseware. 

3. Incentives and Compensation: Instructors were hesitant to learn how to 
maintain and construct Web sites without some sort of incentive or 
compensation. The management and the team did not take the time to find out 
what incentives and compensation the instructors needed. 

4. Unanticipated Areas of Concern: The Web-Design Team did not anticipate 
several areas of concern, such as Internet security and protecting copyrights on 
intellectual property, among others. 

Although the University of Minnesota used an initial decision process, that 
process was inadequate because it failed to identify other issues, such as funding, 
bureaucracy, instructors' opposition and areas of concern mentioned above. However, their 
experience in converting their courses to NBI resulted in lessons learned that are beneficial 
to their future conversion projects. 

Some of the lessons learned are the following: (a) must keep users’ needs in 
mind first; (b) investigate what the university is taking with respect to Internet development; 
(c) consider the ability of instructors to implement and support any Web initiative; and (d) 
to start small and work up to a more ambitious as the development initiative progress. 
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2. Jefferson County Public Schools 

['he following is the assessment of the decision process used by Jefferson County 
Public School District: 

a. Identify Goals 

The Jefferson County Public Schools were interested in introducing K-12 
on-line educational alternatives into their existing district-wide curriculum. Their intention 
was . to better serve students and faculty by designing an alternative course that would 
reach all the populations served by the district. [Ref. 26] 

b. Needs Assessment. 

Their first step was to conduct a formal needs assessment. The assessment 
was undertaken with the hopes of exploring and discovering academic directions that would 
enable the district to expand its educational vision and implementation and enable more 
students to successfully complete their high school education. 

Based upon their initial assumptions and findings, they recommended the 
introduction of NBI alternatives as the most appropriate distance learning direction. They 
also conducted a cost-effectiveness analysis. Their goal was to design and deliver an 
alternative that would benefit all the populations served by the district. 

As a result of their assessment, they formed an Initial Design Team. Their 
design team was comprised of self-recruited individuals who were able to bring critical 
expertise to the product development. These volunteers consisted of discipline-specific 
experts, Web-design professionals, program administrators, standards-based education 
consultants, and educational technologists. In addition to the input of people within their 
district, the school began closely working with outside agencies, industries, and 
organizations. Their reason for initiating this working relationship was twofold: (1) to 
ensure that work-related realities were introduced into their educational content; and (2) to 
strengthen their program base for future expansion and possible working partnerships. This 
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diverse mixture of knowledge ensured a strong foundation upon which to build and expand 
the NBI conversion initiative. 

c. Establish Criteria 

In this example, the Design Team established and fulfilled the criteria set 
forth by a number of educational conversion programs underway at different levels in the 
district. They established the following criteria for instructor selection: (1) appropriate 
content knowledge and experience; (2) competence with technologies required to deliver 
the course successfully-i.e., e-mail, real-time communications, group mailing lists; (3) 
ability to introduce and manage both synchronous and asynchronous communications and 
interactions; and (4) openness to emerging technologies and ways to apply them to online 
educational environments. [Ref. 26] Selection criteria for course/module were not 
addressed, and vendor selection criteria were not applicable. 

d. Selecting Module , Instructor and Vendor 

The Administrator and the Web Design Team selected the instructor and 
course to convert according to their established criteria. 

e. Planning the Project 

During the planning of their project, the team identified the standards and 
expectations. They were able to outline the requirements for their initial on-line course 
design and development. It was this step that enabled them to know, from the outset, the 
challenges ahead of them and ways to go about dealing with them. 

They successfully developed a single Web-based course. In addition, they 
were attempting to integrate district content standard, alternative assessment formats, the 
latest technology, and sensitivity toward gender and culture. This is an example of an on- 
line program put together thorough clear and cohesive vision, careful planning, and 
teamwork. 
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E. ASSESSMENT OF THE IDEA (NPS) DECISION PROCESS 

The following are assessments of the decision process used by IDEA in the 
conversion of these two EME modules to NBI using the Initial Phase of the ISD Model: 

1. Identify Goals 

The NBI conversion decision process began when the proposal IDEA submitted to 
NSHS was approved and funded by BUMED. Their goal is to convert traditional 
classroom instruction to EME modules to NBI. This initiative will provide senior DoD 
medical executives an opportunity for anytime, anywhere EME education on demand on 
both synchronous and asynchronous environments. 

2. Needs Assessment 

The IDEA management and the Distributive Learning Team discussed the project in 
detail and arrived at a consensus on how the development process would proceed. Based on 
this assessment, they agreed that they did not have the required knowledge and experience 
to do the conversion project in-house and recommended outsourcing the Web Design 
Teams. 

3. Criteria for Selecting Modules, Instructors, and Vendors 

The Distributive Learning Team did not establish the criteria for the selection of 
modules, instructors, and vendors for the conversion project. Without established selection 
criteria, it was difficult for the management and the team to arrive at solid decisions in these 
areas. 

The management and the Distributive Learning Team, without previous experience 
in this work, should have conducted a more thorough assessment during this phase of the 
decision making process in order to define the problem and determine possible solutions. 
Part of this phase is the establishment of criteria or standards for selection of modules, 
instructors, and vendors prior to interviewing and selecting contractors. Careful analysis is 
more important when the project is a pilot study, where most of the decision-makers have 
little familiarity with NBI conversion. They should be careful to select only contractors 
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with proven track records and capabilities, who can convert and produce an excellent EME 
module to NBI. 

In addition, there are no established criteria tor module selection. The criteria used 
were from the study by Crawford and Suchan, “Media Selection in Graduate Education."’ 
This study was based on VTE media selection, not NBI. Many additional factors must be 
taken into account when selecting a medium of instruction: type of content, number and 
location of students, time available, instructional budget, and facilities available. Although 
the management and the distance learning team have knowledge of these factors, and made 
good module selections using Crawford and Suchan study; they do not have the educational 
support tools, such as Advisor 2.0, to help determine the feasibility and effectiveness of 
using several alternate media, including NBI. 

IDEA does not have established criteria or standards for instructor selection. During 
the selection process, the management and the Distributive Learning Team selected 
Professor Doyle, based mainly on his success in teaching, as described in Chapter III. Since 
they had chosen Professor Doyle, IDEA must also select his modules for conversion. Thus, 
according to IDEA, it was a good match of professor and teaching modules. This appeared 
to be a favorable coincidence. Presently, the process used by IDEA is to select an instructor 
first and then select his or her respective module. One recommendation is to select the 
module that meets the criteria for conversion and is most beneficial for the senior medical 
executives; and then select the instructor assigned to the module who meets the established 
criteria for instructor. 

As discussed in Chapter III, IDEA does not have a design team and has limited 
programming capability to convert EME modules to NBI. To achieve the conversion of 
EME modules, IDEA had no choice but to outsource to a Web design team. However, 
IDEA does not have established criteria for selecting vendors or contractors. They sub- 
contracted for their Web Design and Development Team and selected the two vendors 
discussed in Chapter III. STG was selected based on previous military training 
development experience, while Arista was selected based on their actual NBI course 
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conversion experience and being located in Monterey County. Although the two 
contractors may be doing the conversion of EME modules satisfactorily, a more effective 
way would have been to establish criteria and standards first and then select the most 
qualified vendor. 

4. Planning the Project 

In planning their project, the IDEA management and Distributive Learning Team 
submitted a well-written proposal to BUMED. Their proposal outlined the phased 
implementation of the distance learning applications within the EME Program. Also 
included were the plan of action, milestones, and deliverables for 1997-1998. However, 
some of the issues, concerns, and constraints on this conversion project were not addressed. 

5. Other Issues and Constraints 

a. Distributive Learning Team Performance 

The process for selecting modules, instructor, and vendors starts with the 
Distributive Learning Team. The team is made up of NPS faculty members and staff whose 
job is to support and assist the management in making a sound decision. One team member 
was not involved in the decision process for vendor selection and did not have knowledge 
on why the vendors were selected. All members must be carefully selected and should be 
included in decisions and work as a team in order to be more effective in the decision 
making process. Their responsibilities include: (1) overseeing and managing the delivery of 
residential and distributed graduate education; (2) creating or adapting content to new 
distributed learning pedagogies; (3) exploring new instructional system technologies to 
foster collaborative learning environments (synchronous and asynchronous); (4) providing 
analysis in determining course/class attributes; (5) facilitating customer support/interface 
between facilities, contractors, and subject matter experts; and (6) conducting cost-benefit 
analyses of alternatives. The team should have identified and evaluated the soundness of 
alternative choices for modules, instructors and contractors before submitting its 
recommendation to the Director and Executive Director of IDEA for the final decision. 
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b. Support for the instructor 

The instructor doing the conversion experienced some problems in working 
with the contractors, including incompatibility of the systems (Mac and IBM), contractors' 
underestimation of the amount of work to be done, and the need to deal with several points 
of contact. [Ref. 29] 

The instructor encountered these problems during the early stages of 
transformation process. Although these problems were later resolved, both the professor 
and the contractor lost valuable time. This could have been avoided with careful advance 
planning, which includes establishing one project manager for each vendor, checking on the 
compatibility of NPS's and the vendor's system, and others. 

c. Funding 

Certainly, one constraint on this project was funding. The decision to 
implement a particular instructional program or change an existing one rests on identifying 
all the costs of research and development, personnel, and running the implementation for 
the life of the system, including operation and maintenance. Also, conversion of modules 
to NBI requires an up-front investment while resident training continues. With the $2.5 
million funding for the EME program, which includes regular classroom training and 
conversion of EME modules to NBI and VTT, only a small portion of the funding is slated 
for research and development in the conversion to NBI. According to IDEA, BUMED is 
planning to convert about 28 EME modules to NBI. On average, it will cost $100,000 to 
convert each module to NBI; with 28 modules, this will amount to $2.8 million. A limited 
budget will definitely restrict further NBI conversion projects. 

(L Instructors ' Opposition 

Another constraint is the instructor. Web-based instruction may be an 
excellent new tool for learning and teaching, but faculty members face many obstacles to its 
effective integration into instruction. Some faculty members are against the conversion of 
EME modules to NBI and the used of other distance learning media. They may be 
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unwilling to participate in the conversion process. Critical to the future of every tenure- 
tracked assistant or associate professor are promotion and tenure criteria. Without a doubt, 
the tenure process serves as a strong incentive for research and an equal disincentive for 
instructional innovation. As a result, junior faculty might believe it is not in their best 
interest to participate in Web-based instruction initiatives unless promotion and tenure 
criteria reward instructional innovation. [Ref. 30] 

F. SUMMARY 

The successful development and maintenance of a new instructional system, such as 
the conversion of course modules to NBI, requires a systematic decision process. This 
process should include analysis, design, development, implementation, and evaluation of 
the program. Criteria should also be established, especially in the module, instructor, and 
vendor selection process. This will eliminate surprises and non-performance. 

Most first-time projects encounter more frequent problems when poor or inadequate 
decision processes are established and no careful planning is done. Many organizations, 
such as the University of Minnesota, Jefferson County Public Schools, and NPS, learned 
lessons from their experience that will be useful in their future NBI conversion projects. 

IDEA management and the distributive team must recognize the importance of a 
comprehensive decision process in developing new instructional systems. Although an 
initial decision process was used, it was neither fully developed nor comprehensive. The 
criteria for selection of modules, instructors, and vendors were not included in their 
assessment. Additionally, some important issues such as instructor’s compensation and 
incentives, intellectual property rights among others that may affect the future conversion 
process were not addressed. 
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V. CONCLUSIONS AND RECOMMENDATIONS 



I his research examined the decision process behind the conversion of course 
modules trom the Navy Medical Department Executive Management Education curriculum 
to Network-Based Instruction. The objective was to track and model the process of 
integrating NBI technology into an existing traditional classroom course of instruction. The 
goal was to capture the context of the key decisions made during the preliminary stage of an 
actual conversion project and analyze the effectiveness of the approach. 

A. RESEARCH QUESTIONS 

1. What Process Was Used to Convert the Current EME Modules to NBI? 

As discussed in Chapter IV, IDEA used a process that is not fully developed in the 
initial decision of converting current EME modules to NBI. This undertaking should be 
based on reasonable expectations regarding the probable outcomes or benefits from such an 
endeavor. The IDEA management and Distributive Learning Team should use models such 
as Instructional Systems Development to plan, develop, and deliver education and training 
programs. 

2, Who Are the Stakeholders? 

As discussed in Chapter II, a stakeholder is any group that is affected by the 
achievement of an organization’s objectives. To be a stakeholder means to have an interest 
in the process or outcome of the issue. Some stakeholders may want things to stay the 
same; some may favor small changes of various sorts; some may want radical change. 
There may be stakeholders who are not involved in any controversy or process, yet have an 
interest in the issue. Active stakeholders are organizations or groups or occasional 
individuals who seek to affect the current initiative because of some interest. Some of them 
may be public-spirited or acting on behalf of others. 
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The EME initiative has six stakeholders, not all of whom are equally affected by or 
have a significant effect on the initiative: 

1 . Commanding Officers and senior military personnel are the most significantly 
affected by this initiative. They are required to complete the EME curriculum in 
order to acquire necessary proficiency and demonstrate professional 
administrative skills. 

2. Congress is also an important stakeholder because, as the primary decision- 
maker, it issued a directive to the DoD concerning the required skills, 
knowledge, and experiences an MTF commander must possess. 

3. Clearly, DoD, including the individual military services, has a major stake in 
this EME initiative. The DoD's position is that the training of senior medical 
personnel is vital for health care executives to achieve the necessary skills. 

4. Professors who are experts in their respective fields comprise another group that 
can exert influence. The obvious impact of this initiative is that, as more 
medical officers go through this program, more professors are needed to teach 
the required skills. 

5. BUMED has a direct impact on the EME initiative. They are involved in the 
design and funding of the EME program based on the competencies identified. 

6. NPS Institute for Defense Education and Analysis (IDEA) is charged with the 
development, delivery and management of EME programs. 

3. What Is the Perceived Value of the Conversion to NBI? 

The conversion of EME modules to NBI will provide an opportunity for anytime, 
anywhere EME education on demand in both synchronous and asynchronous environments. 
Also, successful conversion of EME modules to NBI may eventually become a revenue 
generator for NPS. The potential benefits are increased training effectiveness and savings 
of travel and per diem costs for resident training. As discussed in Chapter II, training 
effectiveness is a complex outcome measure. It includes the ability of NBI to provide at 
least the same amount of knowledge and as well-educated of a learner compared to the 
traditional classroom, quicker ability to update course materials, and increase throughput of 
the number of students. 
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4. What are the Significant Resources and Constraints? 

One of the resources and constraints on this project is funding. The decision to 
implement a particular instructional program or change an existing one rests on identifying 
all the costs of research and development, personnel, and life cycle costs, including 
operation and maintenance. Also, conversion of modules to NBI requires an up-front 
investment while resident training continues. 

Another potential constraints are instructors. According to IDEA, BUMED plans to 
convert 28 modules to NBI, finding all 28 qualified instructors could be difficult. Faculty 
members are faced with many obstacles to its effective integration into NBI instruction. 
Some faculty members are against the conversion of EME modules to NBI and the use of 
other distance learning media. Because of the publishing demands of a tenure-track 
position and other duties, it is difficult for professors to spare the time and help the Web 
Design Team in providing his or her input to their modules’ design, development, and 
implementation. Even when the conversion of module is being outsource to private 
contractors, this still requires a lot of instructor’s time to work with vendor. Also, some 
instructors are poorly prepared and have no knowledge in latest computer technologies. 
During the implementation stage, it would be helpful for the instructor to possess a broad 
range of computer skills to deliver the course successfully--!. e., e-mail, real time 
communication, and group mailing lists. If faculty members must acquire any of these new 
skills in order to carry out their vision, entire weeks and months can be swallowed without a 
significant product to demonstrate. Another reason for instructors objection to EME 
conversion is the lack of incentives and compensation. 

B. CONCLUSIONS 

As the Information Age evolves, and technical advances make resources more 
accessible, NBI is becoming a more viable medium to facilitate learning. Several 
organizations, including the DoD, schools, and private companies, have adapted to this 
changing technology. NBI makes it easier and quicker for organizations to train and 
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educate their personnel anytime and anywhere. However, conversion of EME modules to 
NBI requires a large investment up-front. Transformation of more modules will reduce the 
number of training days, resulting in greater savings. The potential benefits are increased 
training effectiveness and saving of travel and per diem costs for resident training. 

Developing and maintaining a new instructional system, such as the conversion of 
course modules to NBI. requires a decision process. This process should include analysis, 
design, development, implementation, and evaluation of the program. Criteria should also 
be established, especially in the selection process. This will eliminate surprises and non- 
performance. 



C. RECOMMENDATIONS 

Based on this research, the following recommendations are made: 

1. Continue the conversion of EME modules to NBI. The more modules are 
converted the more cost effective it will be. Also, converted EME modules can 
reach the greatest number of students anytime and anywhere. 

2. Establish a proven decision process such as the ISD Model to plan, develop, and 
deliver educational programs. This will help assists in fulfilling the 
requirements for the successful implementation of educational and training 
programs. 

3 . Provide training to professors who are willing and interested in converting their 
EME modules in the future. 

4. Establish compensation and incentives to attract and create interest among 
professors in converting EME modules to NBI. 

D. AREAS FOR FURTHER RESEARCH 

The conversion of EME modules to NBI raises possibilities for future research. 
These include: (1) life cycle maintenance and upkeep of the program (2) marketing and 
implementation; (3) intellectual property rights; and (4) empirical testing to determine the 
effects, number of sites, students, and different learning outcomes. 
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APPENDIX FEASIBILITY STUDY PRODUCED BY ADVISOR 2.0 



BASIC INFORMATION 

Date: 

Course Title: 
Course Number: 
Course Manager: 
Operator: ~ 

Date of Last Audit: 



04-17-1998 

Budgeting for Defense Health 
Module 24 

Professor Richard Doyle 
LT Cesar A. Odvina 
04-17-1998 



ENVIRONMENT 



REGULATIONS 

Regulations 

Reasons for Course: 
Classification: 

Testing Requirements: 

ORGANIZATIONAL 

Administrative 

Development / Revision Time: 
Trainers / SME Availability: 
Reference Material Availability: 
Consistency: 

Delivery Time: 

Scheduling: 

Data Collected: 



Course 

Objectives: 

Content: 

Status: 



Combination 
Public Domain 
Skill / Knowledge 



Adequate 

Adequate 

Accessible 

Important 

Flexible 

Meets demand 

Large 



Job performance 

Specific 

New 



Trainees 

Size: 

Location: 

Value of Trainees Time: 



>1000 
Scattered 
Very high 
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TRAINING I 



Trainers 



Delivery Method: 


SME 


Data Collected: 


All of the above 


Resistance to Change: 


Medium 


Trainees 




Team Work: 


No 


Evaluation: 


Skill / Knowledge 


Course 




Relevance: 


Prerequisite 


Content: 


All of the above 


Referred To: 


Regularly 


Expected Life (years): 


> 5 years 


Stability (%): 


5% to 10% 


Urgency: 


Moderate 


Length (hours): 


4 hours 


TRAINING II 




Content 




Hands on Exercises: 


No 


Personal Safety: 


No 


Equipment / Data Integrity: 


No 


Real Equipment: 


Required, available 


Real Equipment Simulation: 


Moderate 


Processor in Real Equipment: 


Not applicable 


Guided Discussions: 


Desired 


Role Play: 


No 


Teaming Exercises: 


No 


Audio: 


Required 


Full Motion Video: 


Desired 


Still Images: 


Required 
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INSTRUCTIONAL 



Trainees 



Receptive to computers: 


Yes 


Reading Ability: 


Adequate 


Resistance to Change: 


Medium 


Motivation: 


Medium 


Skill / Proficiency: 


Diverse 


Ability to Travel: 


Limited 


Access to Computer: 


Anytime 


Access to Multimedia PC: 


Anytime 


Access to Video Conferencing: 


Limited 


Access to Internet / Intranet: 


Anytime 


Content 


Learning objectives: 


Knowledge / Skill 


Self Pacing: 


Most of course 


Share Experiences: 


Not required 


Apply Knowledge: 


Complex 


SUMMARY 


Delivery Methods / Media 


Scores 


Multimedia CBT 


94% 


EPSS 


87% 


Embedded Training 


84% 


Hypermedia 


83 % 


Internet / Intranet 


83% 


Video Tapes 


72% 


On-location Seminar 


67% 


Audio Tapes 


63% 


Video-Tele Conferencing 


60% 


Video Conferencing 


60% 


Classroom 


58% 


Computer Conferencing 


57% 


Audio Conferencing 


50% 


Simulators / Virtual Reality 


0% 


Customized CBT 


0% 


Intelligent Tutorial 


0% 


Off-the-Shelf CBT 


0% 



51 



Hypertext 


0% 


Print 


0% 


Simple CBT 


0% 
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